Making a Shallow Network Deep: Growing a Tree from Decision Regions of a Boosting Classifier

This paper presents a novel way to speed up the classification time of a boosting classifier. We make the shallow (flat) network deep (hierarchical) by growing a tree from the decision regions of a given boosting classifier. This provides many short paths for speeding up and preserves the Boosting decision regions, which are reasonably smooth for good generalisation. We express the conversion as a Boolean optimisation problem. Boosting as a tree: A cascade of boosting classifiers, which could be seen as a degenerate tree (see Figure 1(a)), effectively improves the classification speed. Designing a cascade, however, involves manual efforts for setting a number of parameters: the number of classifier stages, the number of weak-learners and the threshold per stage. In this work, we propose a novel way to reduce down the classification time of a boosting classifier not relying on a design of cascade. The chance for improvement comes from the fact that a standard boosting classifier can be seen as a very shallow network, see Figure 1(b), where each weak-learner is a decision-stump and all weak-learners are used to make a decision. Conversion of a boosting classifier into a tree: Whereas a boosting classifier places decision stumps in a flat structure, a decision tree has a deep and hierarchical structure (see Figure 1(b) and 2(b)). The different structures lead to different behaviours: Boosting has a better generalisation via reasonably smooth decision regions but is not optimal in classification time. Whereas a conventional decision tree forms complex decision regions trying classification of all training points, a boosting classifier exhibits a reasonable smoothness in decision regions (see Figure 2(a)). We propose a method to grow a tree from the decision regions of a boosting classifier. As shown in Figure 2(b), the tree obtained, called super tree, preserves the Boosting decision regions: it places a leaf node on every region that is important to form the identical decision boundary (i.e. accuracy). In the mean time, Super tree has many short paths that reduce the average number of weak-learners to use when classifying a data point. In the example, super tree on average needs 3.8 weak-learners to perform classification whereas the boosting classifier needs 20. Boolean optimisation: A standard boosting classifier is represented by the weighted sum of binary weak-learners as H(x) = ∑i=1 αihi(x), W2